7.5. Online Evaluation

How is online evaluation different from offline evaluation?

Offline eval runs a versioned dataset before release. Online eval samples traces generated by real traffic after release and scores them asynchronously. It detects input/behavior drift the fixed set did not anticipate, but it also handles more sensitive data and cannot block a response after the fact.

Does the course ship an automatic online evaluator?

No. It ships trace collection, MLflow storage, code/optional-judge scorers for the fixed eval set, and human trace feedback. It does not schedule a scorer over live traffic, and therefore makes no claim that drift detection is active.

This is a deliberate boundary: a safe online pipeline needs sampling, consent/retention, reviewer access, judge budgets, deduplication, alert thresholds, and incident ownership.

How do you inspect a bounded trace sample?

The shipped MLflow entrypoint names experiment id 0 as ops-copilot, so the runtime collector and named evaluation runs share this query target.

cd agents/python
MLFLOW_TRACKING_URI=http://localhost:5000 \
uv run mlflow traces search \
  --experiment-id 0 \
  --max-results 20 \
  --no-include-spans \
  --output table

Use filter/order options to select a time window, model/revision, error state, or already-assessed subset. Start with metadata-only results; fetch full spans only for authorized investigation.

What would a production scoring job require?

1. Select a representative, rate-limited sample with a documented inclusion rule.
2. Redact/minimize data before any external judge call.
3. Version the scorer/judge prompt/model and record cost/errors.
4. Write assessments back to the source trace without changing it.
5. Alert on sustained statistically meaningful regression, not one low score.
6. Route confirmed issues to an owner and promote sanitized cases offline.

Run scoring outside the request path so judge latency/failure cannot break user traffic.

How do you detect drift without inventing a metric?

Compare distributions by deployment/model/prompt version: tool sequence, error ratio, latency, call count, retrieval no-match, guardrail rejection, and human/scorer assessments. Define a baseline and minimum sample size first. A dashboard line moving is not automatically drift, and absence of a metric is not evidence of stability.

What is the online-evaluation checkpoint?

Write a design for one sampled scorer: exact filter, data fields, redaction, scorer version, cost cap, threshold/window, owner, and rollback action. Execute only the metadata search above; do not enable an automated live judge until privacy and budget approval exist.